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Abstract 


This special report summarizes the results of the CMMI® (Capability Maturity Model® Inte¬ 
gration) Workshop held May 7-8, 2002. The CMMI workshop focused on two types of or¬ 
ganizations: commercial software organizations and information systems organizations. The 
puipose of the workshop was to identify for these types of organizations: (1) barriers to the 
adoption of CMMI, (2) benefits of CMMI adoption, (3) issues created by the migration from 
Software CMM (SW-CMM®) to CMMI. The results of this workshop are collections of 
comments concerning these defined topics regarding CMMI. No interpretation of these 
comments is provided. 


CMMI, CMM, and Capability Maturity Model are registered in the U.S. Patent and Trademark 
Office by Carnegie Mellon University. 
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1 Introduction 


The CMMI (Capability Maturity Model Integration) Workshop on adoption barriers and benefits 
for commercial software and information systems organizations was held May 7-8, 2002, at the 
Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania. The 
workshop was co-sponsored by the Software Engineering Institute (SEI) and the International 
Software Research Institute (ISRI) of Carnegie Mellon University. 

The objectives of the workshop were to better understand: 

• the CMMI adoption needs of commercial software and information systems organizations 

• the barriers and benefits of CMMI for these organizations 

• issues that arise when migrating from Software CMM to CMMI 


The sponsors of the workshop were Bill Peterson (SEI) and Jane Siegel (ISRI). The team that 
coordinated the workshop consisted of Mike Konrad (team leader), Mary Beth Chassis, Mark 
Paulk, and Bill Curtis. Bill Curtis was unable to attend. Gibbie Hart, CMMI Transition Team 
lead, was an observer. Participants attending the workshop were not all from commercial soft¬ 
ware and information systems organizations targeted by the workshop. Participants in the work¬ 
shop were: 

• Richard Basque, Alcyonix 

• Richard Bechtold, Abridge Technology 

• Joseph Billi, Automatic Data Processing (ADP) 

• Venkat Gopalan, DynCorp 

• Subrata Guha, Satyam Computer Services Ltd. 

• Keith Heston, Accenture 

• Gargi Keeni, Tata Consultancy Services (TCS) 

• Dennis Linscomb, DynCorp 

• Mark Servello, ChangeBridge, Inc. 

• Prabhuu Sinha, Satyam Computer Services Ltd. 

• Bob Thorman, L-3 Communications Integrated Systems 

• Joe Youmans, Youmans Consulting 
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Others who contributed position papers for the workshop include: 

• Ruth Berggren, Electronic Data Systems (EDS) 

• Douglas A. Ebert, McKesson Corporation 

• Mutsumi Komuro & Fumihiro Tsunoda, Hitachi Software Engineering Co., Ltd. 

• Don O’ Neill, Don O’ Neill Consulting 

Those interested in the workshop topic, whether or not they were able to attend the workshop, 
were invited to submit position papers, which are summarized in Section 2 of this report. Issues, 
whether benefits or barriers to CMMI adoption, were loosely grouped into 13 categories, and the 
workshop participants multi-voted on the importance of the categories prior to arriving in Pitts¬ 
burgh (multi-vote totals are shown in parentheses after topic title): 

• Emphasis on Systems Engineering (16) 

• CMMI Transition (10) 

• Expanded Coverage (9) 

• Representations (8) 

• Emphasis on Product Lifecycle (6) 

• Assessments (6) 

• Lack of Data (5) 

• Investment/Cost (4) 

• Terminology (3) 

• Limited Coverage (1) 

• Training (1) 

• CMMI Intended for Large Organizations (0) 

• CMMI Business Results (0) 

The process followed during the workshop was to discuss topics by category, as ordered by multi¬ 
vote importance. Using the topics to spark discussion, each participant, selected in random order, 
identified their most important issue regarding CMMI adoption within a category. Three rounds 
of brainstorming per topic identified a number of issues, which were captured on flip charts. A 
participant was allocated two minutes to discuss an issue. After the third round of brainstorming, 
a group discussion elaborated on the issues within the category. The results of brainstorming and 
the discussion are captured in the tables throughout Section 3 of this report. Participants then 
multi-voted on the most important issues within the category. 

After brainstorming through all 13 categories, the issues identified were ordered by their multi¬ 
votes within each category, and another multi-vote was held across all categories. This ordered 
list is in Section 4. The ordered list of issues was then used as a basis for further discussion and 
recommendations, as contained in Section 5. 
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2 Position Paper Topics 


All workshop participants submitted position papers related to the purpose of the workshop. Is¬ 
sues identified in these position papers, whether benefits or barriers to CMMI adoption, were 
loosely grouped into 13 categories in Table 1. The topic category is stated in column one, “Cate¬ 
gory.” Column two, “Issue as Identified in Position Papers,” contains summary statements of the 
issues (as identified in the position papers). 


Table 1: Position Paper Topics 


Category 

Issues as Identified in Position Papers 

1. Emphasis on 

Systems 

Engineering 

a) With appropriate interpretation, CMMI provides a significant 
benefit to information technology and information systems 
(IT/IS) organizations by focusing attention on the “systems” 
aspect of IT application portfolios and technical environments. 


b) Many of our projects are software-only projects and references to 
systems engineering are felt to be largely irrelevant to them. 


c) Many of our projects involve significant hardware issues and can 
benefit from a more rigorous look at systems engineering 
practices. 

2. Emphasis on 

Product Lifecycle 

a) CMMI can be interpreted to focus on new product development, 
where most IT/IS organization workloads are based on existing 
product enhancement/maintenance, causing resistance to usage 
of the model as a tool in process improvement. 


b) CMMI addresses the full life cycle of the product, which benefits 
both commercial software and IT/IS organizations by helping 
them to manage application portfolios across a long period of 
time. 

3. Terminology 

a) CMMI uses systems engineering terminology that is unfamiliar 
to typical IT/IS personnel, leading to resistance to use and slower 
realization of value. 


b) Changes in terminology are not supported by CMMI transition 
material. 
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Category 

Issues as Identified in Position Papers 

4. Expanded 

Coverage 

a) CMMI addresses business value and stakeholder involvement, 
critical issues for commercial software organizations that have 
unclear “customers,” and IT/IS organizations where involvement 
of stakeholders is not typically done in an effective manner. 


b) There are too many process areas (PAs) in CMMI, which results 
in overlap and cumbersome assessments. 


c) The new model, besides its huge size, has complicated 
architecture and representations. 


d) Many groups within our organization have resisted using any 
external process model; the depth and complexity of CMMI 
causes even more resistance from these groups. 


e) CMMI is an improvement over the Software CMM for 

organizations that acquire and use COTS and are responsible for 
acquiring and using system software and hardware. 


f) The CMMI model lays greater emphasis on the engineering 
activities and how a project is done. 


g) CMMI has a new PA, Measurement and Analysis at Level 2, 
which details the measurement objectives, practices, and results 
of a project and aligns them to the organization’s needs. The 
move toward having an integrated measurement system that 
meets management information systems (MIS) needs at Level 2 
forces early focus on the business case, which is very necessary, 
but was omitted in the Software CMM. 


h) Decision Analysis and Resolution (DAR) is a new PA that 
discusses structured decision making using established criteria 
and appropriate situations for the same. 


i) Risk Management (RSKM) is addressed as a separate PA, while 
it was included in the Integrated Software Management key PA 
(KPA) in the Software CMM. 


j) Software Product Engineering (SPE) in the Software CMM has 
been split into Requirements Development (RD), Technical 
Solution (TS), Product Integration (PI), Validation (VAL), and 
Verification (VER). Verification also covers the peer reviews 

KPA in the Software CMM. 


k) Product evaluation criteria are explicitly specified in the Process 
and Product Quality Assurance (PPQA) PA, which was missing 
in the Software CMM. 
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Category 

Issues as Identified in Position Papers 


1) There seem to be some PAs not too relevant for software 

development, at least in our company. (Decision Analysis and 
Resolution [DAR] and Product Integration [PI]) 


m) The CMMI Framework is too large to easily understand and 
SCAMPI sm assessments take a long time. 


n) CMMI addresses a number of issues that the CMM, when 
applied literally, misses. 


o) All of the practices in CMMI appear to be good ideas that an 
organization can use as the foundation for an improvement 
program. 

5. Limited Coverage 

a) CMMI does not address very well IT governance from the 

perspective of the Board of Directors, thereby making it difficult 
to sell the use of CMMI to the CEO for company-wide use. 


b) CMMI does not contain an IT security management component, 
thereby making it difficult to use in IT security planning and 
auditing, which are significant issues especially in relation to the 
use of commercial software products. 


c) CMMI is mainly applicable and advantageous where a number of 
groups are involved in the development. It may be of particular 
use in large projects which involve multi-disciplinary teams 
working together. 


d) Not all “pre-project activities” are included in PAs. 


e) Guidance has not been provided to allow adaptation of CMMI to 
software-only organizations. 

6. Lack of Data 

a) Case study data from early adopters of CMMI is not documented 
and made available as it is for the Software CMM. 


b) A lack of clear “cost of ownership” for adopting CMMI software 
process improvement efforts leads to wholly unrealistic 
expectations from senior staff. 


c) Data regarding the return on investment (ROI) and other gains 
and benefits experienced by organizations engaged in system 
integration and following a CMMI model are not available. 


d) ROI data from CMMI use has not been yet validated whereas the 
Software CMM has a proven track record. 


SM SCAMPI is a service mark of Carnegie Mellon University. 
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Category 

Issues as Identified in Position Papers 

7. CMMI Intended 
for Large 
Organizations 

a) CMMI gives the impression of being only applicable to large 
organizations that are capable of making an up-front, sizeable, 
long-term investment. 

8. Training 

a) A lack of good “getting started” training classes hinders early 
adopters—especially those with a limited formal Software CMM 
or SE-CMM background. 

9. CMMI Business 
Results 

a) CMMI does not include any critical business success areas 
(versus management and technical areas). 

10. Representations 

a) Due to the continuous representation, CMMI is more amenable 
to incremental implementation. 


b) The CMMI continuous representation is ambiguous in the 

definition of certain PAs and the assignment of capability levels 
(CLs) to those PAs. 


c) The two CMMI representations (staged and continuous) make it 
difficult to compare different organizations with respect to their 
process improvement accomplishments. 


d) CMMI can be implemented using a process capability approach 
(continuous model) or an organizational maturity level approach 
(staged model). Organizations opting for CMMI with no history 
of the Software CMM may prefer to implement the continuous 
model since it enables them to work on their selected focus areas. 


e) The dual representations of CMMI, combined to the progressive 
and rapid inclusion of several disciplines, makes it very 
challenging for organizations using CMMI (as “users” or as 
“consultants”) to catch on easily, leading to a tendency to slow 
down, have more anxiety, and negative perceptions. 


f) Anything that shifts the focus from maturity level scores to true 
process maturity is positive. 


g) The CMMI continuous representation facilitates relating process 
improvement efforts to the business objectives of commercial 
market-driven companies. 

11. CMMI Transition 

a) Transition from the Software CMM to CMMI may not be 

relevant for parts of the organization that are primarily involved 
in maintenance of software or small development projects. 


b) The current (i.e., based on the Software CMM) processes work 
well and benefits from them are evident. 
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Category 

Issues as Identified in Position Papers 


c) The transition path for companies moving from the Software 

CMM to CMMI is not clear. Approaches along with tools and 
training to achieve the transition are not evident. 


d) Although the ultimate goal of achieving more business at a profit 
is the same for market- and contract-driven companies, achieving 
“certified” Software CMM and CMMI goals can be more 
directly tied to qualifying for and winning Department of 

Defense (DoD) sponsored contracts in the contract-driven world. 


e) The overall business profile of our company has remained the 
same and guidance provided by the Software CMM is adequate. 


f) Our organization has also adopted the P-CMM (People CMM). 

A number of requirements of CMMI (e.g. Integrated Teaming 
and Organization Environment for Integration), which are not 
addressed by the Software CMM, have been taken care of by the 
P-CMM. 


g) CMMI being published in English only makes its adoption 

challenging for countries where English is not the first language. 
Even when the organizational policy (in multi-national 
corporations) is to adopt English, in the trenches, where 
resistance to change is the most prone to develop, people do not 
necessarily speak English fluently enough to thoroughly 
understand the model. 


h) Without proper project management, the complexity of a CMMI 
process improvement program will generate confusion and 
rebellion (pillars of wisdom). 


i) Higher maturity practices of the Software CMM provide a 
process improvement program for CMMI. (Crawl before you 
walk.) 


j) We have found that providing the process elements, within the 
proper ownership structure, naturally creates participation. 

Senior management leadership of the process improvement 
program and the policy architecture to communicate expectations 
led by CMMI-based services is the mutually beneficial way 
toward enterprise participation. 


k) The CMMI initiative is software driven (e.g. SEI & ownership 
issue). 


1) There is no market pressure on our company to implement 

CMMI. 
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Category 

Issues as Identified in Position Papers 


m) When users actually use CMMI to concretely deploy integrated 
software/systems engineering practices for a sustained period of 
time, the barrier between the software and system communities 
will progressively fall. 

12. Assessments 

a) The SCAMPI (1.02) assessment is not a burden for the assessed 
projects. 


b) The fact that SCAMPI 1.1 uses process improvement indicators 
(PIIs) and that SCAMPI is now being used for BOTH internal 
(like a CMM-Based Assessment for Internal Process 

Improvement [CBA IPI]) and external (like a Software 

Capability Evaluation [SCE sm ]) goals, it will be particularly 
challenging to maintain the motivation among the CMMI 
community to pursue continuous internal process improvement. 
There is a serious risk of falling in the “minimal conformity” 
paradigm (i.e., the “legalistic” approach). 


c) The lack of an incremental transition path from the Software 

CMM and CBA IPI to the CMMI and SCAMPI is a barrier to 
acceptance and incorporation of CMMI, especially by the 
commercial business community. 


d) A recognized and established mechanism for assessing 

incremental progress using a CMMI model with a continuous 
representation will facilitate CMMI model use in commercial 
businesses. 

13. Investment/Cost 

a) Excessive fees for lead assessor licensing and fees have 

significant impact on an enterprise of our size. The five-year fee 
structure for conducting CMMI assessments across our 
organization will approach $350,000. This is significantly higher 
by hundreds of thousands of dollars than for support of Software 
CMM assessments. 


b) A corporate license or fee limit that would cap annual costs for 
Introduction to CMMI for assessment teams, annual renewals, 
royalties, and observations would enable us to continue to 
perform SCAMPI assessments. 


c) Investment has been made in adopting the Software CMM model 
and a mechanism for maintaining the same exists. An additional 
investment will be required for the transition, but the added value 
of that transition may not be commensurate with the investment. 


d) Sunsetting of the Software CMM does not address the needs of 
those organizations for which making the transition to CMMI 
would be costly and complex. 
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Category 

Issues as Identified in Position Papers 


e) CMMI initiatives are viewed as rework (i.e., don’t pay for the 
same real-estate twice). 


f) It is not clear that investments made in the Software CMM over 
the past few years are preserved in moving to CMMI. 
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3 Brainstorming CMMI Adoption Issues 


The issues identified in the following sections were captured on flip charts during workshop ses¬ 
sions. The flip charts are reproduced here with only very minor changes to ensure the integrity of 
the information is preserved. At the end of each session, which covered a particular issue cate¬ 
gory, participants were asked to vote on the most important issues. These votes are captured and 
represented in the left column of each table. 


Before the brainstorming activities, the participants were first asked if they had made a decision 
to adopt CMMI. A “yes” answer indicates that the participant’s organization has decided to adopt 
CMMI. A “no” answer indicates that the participant’s organization has not yet made a decision or 
has decided not to adopt CMMI at this time. Table 2 contains each participant’s response to the 
yes/no question and the rationale behind their organization’s current position. 


Table 2: Decision to Adopt CMMI 

Decision to Adopt CMMI 

Yes. Level 3, SW-CMM proved useful inside the company, but work with many systems 
engineering, COTS, systems integration companies—getting ready for class C assessments. 

No. Not clear what the benefits are. 100 internal organizations, software-only organizations. 
Systems engineering is not applicable. Mainframe COBOL. SW-CMM works, better 
performance out of IT organizations. Big investment in SW-CMM, perceived loss of initial 
investment. Why? Still making progress. 

Yes. Class C SCAMPI. CMMI lots of lift. Primarily a systems integrator, huge benefit because 
of integrated approach. World picture, rework problem, selling is no problem. Implementation 
and deployment is a challenge. 

No. Our target is the same. Have ISO. What is wrong with SW-CMM? Why can we not 
continue? Sunsetting put pressure to re-evaluate. Difficulty seeing benefit with CMMI. Talked 
to customers. Cost is high going to CMMI. Will not move even if SW-CMM sunset. 

No. CMMI is not on our short list. ISO 9000, SW-CMM, and e-Services Capability Model are. 
Within our company: (1) Government contracting organizations have adopted CMMI, consistent 
with what they already do, no big leap. (2) Outsourcing groups—no benefit, does not meet our 
priorities. Change was from walking to driving a car when going to SW-CMM. With CMMI, it 
is just another car. 

Yes. 1. Realized that sponsors usually want to have benefits spread across the organizations. 2. 
Realized there were activities not covered with SW-CMM. 3. Needed evolution of best 
practices. 4. Our company is involved with ISO 15504, and CMMI is aligned. 
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No. One government agency will use SW-CMM regardless of sunsetting, accept staged 
framework of CMMI for other organizations. Success of SW-CMM is quite visible, little 
motivation to retire something that works. CMMI—implement one PA is a benefit, but can you 
really do this? 

Yes. We believe in process improvement. CMMI is an evolved model that addresses many 
issues (terminology, scope, and focus on an area). Customer perspective—glass is Vi empty, 
glass is Vi full. We can use CMMI to bring other parts of the organization that can benefit— 
other side is that their investment in SW-CMM will be lost. 

Deal with small, market-driven companies trying to figure out a way to do the transition without 
a lot of cost. 

Not a clear-cut yes or no. We have adopted Software CMM and People CMM. CMMI has 
pockets of use in our organization. Do we have to take the whole organization to CMMI? Even 
where it doesn’t apply? This approach won’t work for us. Investment is a concern, we have 
already built an infrastructure for supporting Software CMM—so we need not adopt CMMI for 
the whole organization, but will pilot it where it fits, to determine the benefits. 


3.1 Emphasis on Systems Engineering 

Table 3 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Emphasis on Systems Engineering.” Column two above the “Discussion” bar contains 
the results of the brainstorming. Column two below the “Discussion” bar contains related points 
made during participant discussion of the issues. Column 1 contains the number of votes for each 
issue/discussion point that indicates its importance to the workshop participants. 


Table 3 

Emphasis on Systems Engineering 

Votes 

Issues 

Brainstorming 

10 

For software-only, how do vou interpret CMMI consistently, if tailoring is left to 
individuals? Wrong approach—need tailoring guidelines. 


Barriers between software and systems engineering will never fall, but will eliminate 
cultural no-no’s. REQM, RD, VER, and VAL help to fix problems. 


Migration from SW-CMM to CMMI may not be worth the cost, especially software- 
only, when you are seeing benefit from SW-CMM. First step to SW-CMM—large 
delta. SW-CMM —> CMMI—small/unknown delta. 

1 

Involvement with systems engineering good, but how do you transition a mature 
software group and an immature systems engineering group without burdening the 
software group? 
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Votes 

Issues 

2 

People in CMMI classes are coming from both communities—there is no such thing as 
a software-only project. 

4 

Expand on previous point. We don’t teach our software folks to look at systems 
engineering issues. However, our systems and environments are more complex. 

1 

What can be developed versus what can be implemented, interpretation is an issue. 

What do I gain from re-interpreting this thing? The population is young (2-4 years). 
Large systems perspective is difficult at this level. 

6 

CMMI more holistic approach—1) huge culture change 2) transition—not having 
proper roadmap 3) roadmap 4) leverage investment —» solve and adoption will occur. 

9 

Agree with software-only. However, emphasis—only so many improvement dollars, 
coordination, painful tasks are covered by SW-CMM, (IC covers systems engineering 
issues in SW-CMM). Don’t own systems people, only software. Interpret CMMI for 
software. 


Some PAs, e.g.. Competency Integration of People CMM, have a different name but 
are covered in CMMI. Due to our People CMM initiative, this is already done and 
does not require additional effort in transition. 


When systems engineering is outsourced—providing guidance is desirable (how to 
interpret CMMI). 

12 

Not just method issue but also model. Continued maintenance of SW-CMM (CMM 

V2) for organizations that are predominately software. 

1 

When applying this to airplanes, CMMI provides the robustness. 

6 

SW-CMM might be adequate instead of adopting CMMI. CMMI is not irrelevant. 
Building on systems engineering aspects. More than 90% of organization would need 
SW-CMM, some of the requirements that CMMI provides already addressed. 

10 

Clients struggle for perfect answer. Shift focus from satisfying the model to using the 
model to identify and address problems in the organization. Extremely useful to have 
implementation guides. (Internal IT departments.) Size doesn’t matter; issues are the 
same. 

5 

Need guidance for defining organization types, need to think about different flavors. 

1 

CMMI represents a framework, and you can apply some part of it. You don’t need to 
apply all of it. 


Good source book. The issue is complexity. When you want to focus on software, 
there are a lot of things to think about: continuous/staged, which PA... 
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Votes 

Issues 

Discussion 

1 

There are only a few exceptions where you wouldn’t use certain specific practices if 
you are a software company. There are amplifications—might need a global guidance 
book. 


Appears there is an evolutionary step between SW-CMM and CMMI with this 
interpretation guide. 

3 

Investment in SW-CMM is reusable in CMMI (think about discussions of maturity 
levels [MLs] - score is important). This is not a technical issue but more of a selling 
issue. 


Understand the benefit of CMMI. The difference between SW-CMM and CMMI is 
not that big. 


What is the benefit versus implication of deploying a new model in the organization? 

3 

Coming full circle that software is important, but now saying systems engineering is 
the most important thing. May be unnecessarily distracted. 


Don’t engineer software, and we keep adding more and more scope, but will we lose 
what is important? 

2 

CMMI first comprehensive business model that links various aspects of business. 

3 

More comprehensive view—so how do you implement this in various levels of your 
organization? 


Without CMMI you will have a single vertical view. 


We are shaping the future (spin doctors). Looking at this holistically does not water 
down. 


Are we looking for something different to keep this competitive advantage? 


Since 9/11, security is a big issue—this is an evolving process. 


Embedded systems and COTS are more complex today—so we need to change. 
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3.2 CMMI Transition 


Table 4 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “CMMI Transition.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 


Table 4 CMMI Transition 


Votes 

Issues 

Brainstorming 

1 

Many projects are maintenance projects, and they think that much of “this stuff' 
doesn’t apply—Implementation guides could describe what applies. Things are not 
applicable or partially applicable—[e.g.,] TS, even if architecture exists, need to 
evaluate it to make sure it will meet what you need. 

1 

Keeping a process focus is a project management challenge. Needs to be addressed by 
training or model. Using CMMI in vour process improvement approach is a huge 
effort. 


What is meant by transition? 

• SW-CMM => CMMI 

• [newbie or other model] => CMMI 

SW-CMM => CMMI—chasing a level number. How will CMMI be written into 
contract language? Contract language is a big item. 


Wherever CMMI is relevant, it can be included in the quality management system 
(QMS). That’s not a problem. But for others, where things are working well in the 
framework of Software CMM, we will end up having non-applicable PAs in trying to 
use CMMI. 

9 

Not having a roadmap. 1) Do I make continuous or staged choice? What is applicable, 
what is not? 2) Scale. There is much more in a shorter period of time, e.g., MA. 3) 

Cost: what is the cost of moving from SW-CMM L3 to CMMI? 

15 

No compelling need. Have SW-CMM, and it is working, and you can’t take it away. 
Worst case: have to develop an appraisal method. 
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Votes 

Issues 

13 

Really five transitions: 

1) New architecture (structure is different) 

2) New software processes 

3) Adding SE 

4) Continuous representation 

5) New assessment method 

If you are going for score, you have to address four transitions => roadmap. 


Global transition issue => language barriers. 


CMM1 is a solution looking for a problem. Historically, software is a problem. 

Systems are failing because of software. 

2 

Most difficult issues transitioning from SW-CMM to CMM1. 

1) Items that will not apply. 

2) Subject to interpretation by assessors. 

3) How quickly can someone say something does apply or doesn’t apply? 


Transition from SW-CMM L2 to CMMI L2 causes an obligatory transition period 
because of institutionalization, deploying new processes, and new transition window. 
There will be a violation of contract unless there is some guidance. 

2 

Arguments about CMMI are similar to arguments from early SW-CMM. Cut ties to 
SW-CMM if you want people to transition; won’t make transition as long as there is a 
status quo alternative. 

6 

We adopted ISO as it was a business need (required to do business in Europe) We 
found that Software CMM is a better framework in explicitly bringing out usage of 
data /metrics to manage. A large number of organizations adopted this. Now having a 
Level 5 is a differentiator in bagging assignments. When someone requires CMMI to 
do business, then it may be a different story. ... Otherwise we need to clearly 
understand the benefits in transitioning to CMMI. 


It may be a good idea to use SCAMPI and see how many “not applicables” you get to 
see how applicable CMMI is in software-only organizations. In context of process 
improvement, the benefits of CMMI should be very clear. If the intention is 
improvement, and CMMI helps in improvement, it will be a natural tendency to move 
to CMMI. 

1 

SW-CMM is at the core of the CMMI Product Suite. You aren’t stalling over. The 
high maturity practices in SW-CMM help you with CMMI. This is not all rework, and 
people need to understand that. 
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Votes 

Issues 

12 

Believe CMMI is international, and there are some barriers. Concerned about strategy 
adopted [by SEI] that this is a barrier. The additional 50% added-on price, location 
barrier—Enormous cost and energy. People from other countries are investing a lot of 
time and money—need more balanced solution. 

4 

Commercial community is less contract driven and more market driven. Process 
improvement associated with (driven by) business goals is better. So exploit 
continuous representation. 

3 

This is not evolutionary change but rather revolutionary change—SW-CMM v2C— 
saw what was going on. With CMMI—new complexity. SW-CMM lighter in 
complexity. Where do 1 get started in CMMI? 

2 

Objective of this transition. For the puipose of improvement, do I always have to 
change model? If process improvement (L5) looks for and addresses opportunity, will 
your own processes address these ever changing conditions? Every transition should 
be viewed as a process improvement transition that adds value to customer. 

8 

CMMI very expensive to adopt because expensive for consultants to be “authorized” ~ 
S40K+, 6K royalties per year. Bigger companies will serve this market, and it will be 
more expensive. Compared to ISO 5x to 8x more expensive. 

Discussion 


Different types of companies: 

1) commercial—self-motivation 

2) defense—business drivers/policy, client driven 

3) consulting—business drivers, client driven 


Transition from ISO wasn’t really a problem (incremental). 


Transition to CMMI many different things. 


CMMI being forced in a short period rather than long (analogy => Windows). CMMI 
is more on an evolutionary path than revolutionary path. 


It is a major change. 


Reacting strongly as change agents as changes are happening to us. 


For companies implementing nothing, SW-CMM is a path of compliance for CMMI. 

But you could get the partial credit. 
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1 

Business drivers are critical. What is the value proposition? 

SW-CMM—quality 

CMMI -adoption of systems engineering discipline in areas like COTS 


Less expensive assessment alternatives needed 


3.3 Expanded Coverage 

Table 5 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Expanded Coverage.” Column two above the “Discussion” bar contains the results of 
the brainstorming. Column two below the “Discussion” bar contains related points made during 
participant discussion of the issues. Column 1 contains the number of votes for each is¬ 
sue/discussion point that indicates its importance to the workshop participants. 


Table 5 

Expanded Coverage 

Votes 

Issues 

Brainstorming 

1 

SCAMPI appears to be very labor intensive, consumes calendar time, staff hours. 

2 

Yes, bigger but areas that were open issues in SW-CMM. But needs to be applied 
carefully. 

6 

Too many PAs and some of the wrong kind, e.g. DAR. More PAs isn’t necessarily bad 
but we need more of the right thing. Examples: What do our auditors look for? Many 
use COBIT. We needed a security management model and did not find security 
content in the CMMI. Therefore, we had to go outside the model and choose ISO 

17799. Bottom line: CMMI needs well-defined criteria and guidelines that drive the 
content and expansion of PAs. 

5 

DAR, TS for example. Expansion was difficult to apply in maintenance environment. 

1 

Need this expansion on SPE [from SW-CMM]. 


The expansion of SPE is relevant in a development environment; however it is not so 
critical in maintenance environment. In a development environment: these processes 
are already expanded in all probability. (Comments already provided in position 
papers.) 

6 

Measurement and Analysis (MA) is good with expanding business imperatives—this 
needs to be emphasized and expanded even more. 
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Votes 

Issues 

4 

Explore expanded coverage assessment (delta assessment that only looks at what is 
new). 


Expansion welcomed, [but] when you don’t have choice to adopt, there is an 
investment required that imposes additional overhead on the company. 

3 

Give guidance in partitioning systems engineering and software capability so that if 
you only want to focus on a particular subset you can. 

7 

Clearly see why model needs to be integrated, but who is helped and who is hurt? 

Only auditor is helped; everyone else is hurt. Disservice to the rest of organization 
who usually does a specific task. 

1 

Process asset ownership is much different in CMMI. 

1 

SPE is an improvement to split it out [into multiple PAs]. 

4 

New approach with process [practice] indicators isn’t as cumbersome. It should help 
to gain significant time during the onsite period 

8 

Greatest shortcoming of SW-CMM is that it was implemented literally—not clear that 
this doesn’t get worse in CMMI. 

5 

Requires a higher level of sponsorship. 

Discussion 

8 

[I thought] SEI would put all of their models into one structure; instead SEI “mushed” 
all of their models together, now more painful. 

1 

This thing is SO BIG, impression is that you have to do it all, and you have to do it all 
right now! Must plan strategy more carefully. 


[The original plan was that] CMMI [would be a] repository of components, and you 
could select a button to get what is needed. Since the organizations have to make the 
button decision themselves, then there will be different interpretations. Consistency of 
assessments is a concern. 


Big bulk of world is using CMMs as yardstick. 


Capability is a range or results, not a prefect predictor. 


How did they (ISO) handle the expanded coverage—it was more evolutionary. No 
change in architecture, based on industry need. 
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Votes 

Issues 


One or two people asked for CMMI, now the masses are expected to have this need. 

3 

If I only assess 80% and say I’m CMMI-compliant, then I’m leaving out critical 
information, not truthful to customer. 


If you win lottery and start company where solutions are based on software, what 
would you pick? CMMI. 


If you bought an [existing] company [that didn’t have serious systems issues], would 
your answer be the same? Yes. 


CMMI manages the business better. 


[CMMI is] about risk reduction/migration so that you can reduce risks to customer. 


Tradeoff will increase cost. 

1 

Implied expansion of support within companies and model requires expanded support 
by industry/SEI. What is being done to help this transition? 

7 

SW-CMM in place takes care of a large number of issues, except customer satisfaction 
and complaints, and this helped with transition from ISO 9001:1994 to ISO 9001:2000. 


ISO same body doing certification, but training is what changed, therefore no major 
change/impact. 


When does the expanded coverage stop? It’s just getting started (business 
opportunities, customer satisfaction). Concerned that this is just the beginning. 


Things that are tolerable now will be intolerable in the future. 


“Glass ceiling” if things keep being added—too large, too complex. 


World works with having a “grade.” 


Is security a product-focus attribute? 

• You need to ensure that this needs to be done. 

• Central component of software development. 

4 

Criteria for expansion are needed. 


To date, CMMI is DoD driven. CMMI now needs to address the needs of commercial 
companies, e.g., business goals and security. 

2 

What needs to be added from commercial world? 
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3.4 Representations 

Table 6 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Representations.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 


Table 6 Representations 


Votes 

Issues 

Brainstorming 


Took quite a while to understand the two different structures. Once understood, it still 
would be tough to explain. Should it be staged, continuous, or both? Good idea, but 
hard to decide which representation. Our company eliminated this by picking staged 
and going with it. 

12 

Shift focus from score to process capability is good. 

3 

“Constageduous” approach is key. Both visions are good, but do we need both 
representations? 


Easier to engage folks in process improvement with a Level 1 available (continuous 
CL1). 


Integrated representation only supports the needs of a very small percentage of the 
organization. As model gets larger, this percentage will get smaller. 

1 

2 

Continuous representation is ambiguous 

1) You can’t look at PAs independently. 

• Requirements Development and (RD) Requirements Management 
(REQM) 

• Measurement and Analysis (MA) 

• Project Monitoring and Control (PMC) and Project Planning (PP) 

2) Not enough elaborations, work products, and examples to show what it means to 
go up a CL. What is CL4 & CL5? 

3) Taking PAs and matching them to CLs doesn’t make sense—CL4 on 

Quantitative Project Management (QPM) and MA. 

4) Doesn’t help with meeting business objectives. 


Still confused about continuous representation. Must be skilled in both to understand 
both representations, and this might be impossible to do. Build expertise—want lead 
appraiser (LA) that is skilled in one representation rather than both. 


CMU/SEI-2002-SR-005 


21 




Votes 

Issues 

3 

SEI needs to look into other views into models, e.g., role based. Keep the staged 
representation. 

[The following text was added after workshop] 

For example, you can define a view (via matrix, table, etc.) that shows what parts of 
the CMMI are applicable to you if you were a programmer, a view for development 
project lead, etc. These views would not require a separate physical document as is 
the case with the Continuous Representation. There is already some commercially 
available software that incorporates role views in helping companies develop 
documentation for CMM compliance. That indicates the need in the marketplace for 
those types of views into the model. 

5 

The first indication that there was a problem with representations was when someone 
uttered “constageduous.” 

1 

Maybe we can get by with one representation. If we had guidance on how to get to the 
other. Do we need two auditing approaches? 

7 

From implementation perspective, which representation gets you results faster (ROI 
data)? 

8 

Easier to propose, manage, reward process-improvement efforts if you can align 
progress with PAs. 

- Easier to engage middle management if you align their processes with smaller 
increments instead of a 1-2 year timeframe (MLs). 


Doesn’t seem that there are any quality checks to ensure that SCAMPI is done 
properly (same issue with CBA IPI). 
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Votes 

Issues 

15 

Keep simple; kill continuous representation. 

[The following text was added after the workshop.] 

Why does our company want to kill the Continuous Representation (CR) when no one 
is forcing us to choose it? Why not choose the Staged [Representation] for our 
company and let others choose the Continuous Representation ? Our answer to those 
questions: 

1. Opportunity Costs: Even though vl.l of the CR has been published, there will be 
ongoing work required by the SEI to handle change requests for the CR, to teach 
classes in the CR, and to add more detail to the CR (in terms of elaboration, 
examples, and typical work products) to show what the higher Capability Levels 

(CLs) really mean for certain Process Areas (PAs). Realizing that the SEI does 
not have unlimited funding, all of this effort could be better spent, in our opinion, 
in investigating, incorporating, and maintaining other model content (e.g. 
security, business objectives, tailoring guidelines) into the Staged Representation 
of the CMMI. 

2. Complexity: By having a separate physical document for the CR with the same 
thickness (and actually more goals) than the staged, the perception in the 
marketplace is that CMMI is overly complex. Even though organizations can 
choose a representation, they still need to study both to determine which one is 
best for them. This will lead organizations, especially those in which agile 
processes are used, to choose other less complex and less voluminous process 
models. The end result, in our opinion, is that the CMMI will lose market share. 

We would not like to see that happen since the CMM, especially Level 2-3 KPAs, 
has proven its value to us over and over again. Also, as a company seeking a 
competitive edge, we would like to go with a model that is the leader in market 
share and recognized by most of our clients. 

3. Unnecessary: The ability to pick and choose PAs individually is already available 
in the Staged Representation and has been since the CMM. The advantage of 
choosing them individually from the CMMI Staged Representation and not from 
the CR is that you can see what each PA is based upon. This makes it easier to 
determine what else you should be doing while trying to implement a specific PA, 
i.e. you should be doing related PAs and those at lower maturity levels. 

3 

With staged, less possibility of rewarding people (heard but disagree), but encourage 
other indicators like goal achievement. Have to establish monitoring approach for 
staged. Don’t encourage people to gauge progress against staged. 

8 

Provides opportunity to acknowledge that some PAs change as they mature. 
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Votes 

Issues 


[The following text was added after the workshop.] 

Deficiencies in the representation that could be resolved by either updating the 
Continuous Representation with clarification OR by killing the Continuous 
Representation. 

1. The Continuous Representation assumes dependencies that are not shown by 
represen tation of standalone PAs going from CLO-5. For example: 

• Cannot do Requirements Development without Requirements Management. 

• Cannot do Measuremen t and Analysis in a vacuum. 

• Cannot do Project Monitoring and Control without doing Project Planning. 

While the Continuous Representation text describes some of these dependencies, 
the Continuous Representation diagram does not support these dependencies. 

2. The Continuous Representation often does not give enough detail and examples 
about what the higher-level CLs would look like for a given PA. The following is 
just one of many examples: Project Planning is a Maturity Level (ML) 2 PA in the 
Staged Representation. In the Staged Representation, there are elaboration, 
examples, and typical work products for the Specific Practices (SPs) and the 
Generic Practices (GPs). However, in the Continuous Representation, the 
elaboration, examples, and typical work products stop with GP2. In other words, 
the Continuous Representation does not contain any elaboration, examples, and 
typical work products beyond what was already in the Staged Representation. 

(This appears to be true for cdi PAs in the Continuous Representation.) Therefore, 
if one wants to know what the Project Planning PA would look like at CL3-5, no 
detail is available other than reading the generic descriptions applicable to all 

PAs. 

Certain CLs for some PAs do not appear logical. For example, what does a CL4 
(Quantitatively Managed) mean for the Quantitative Project Managemen t (QPM) PA ? 
If an organization is doing QPM at all, wouldn’t it already be at CL4? 

Discussion 


When teaching, people do understand the differences between the two representations. 


Allows flexibility. 


Companies who adopted SW-CMM could use it any way they see fit. Therefore, no 
need for two representations. 


Industry loses nothing by just having staged. Multiple representations add nothing 
except complexity. 
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Votes 

Issues 


Advantage of picking and choosing in staged, lets you understand the prerequisite 

PAs. 


Minimizes overhead for SEI, company, etc. 


Difference between two representations is minor (look at pages in books). 

1 

Tend to forget about ISO 15504, which requires a continuous representation. 


Have a preference, but is it up to someone to select? 


Failure rate of SW-CMM is greater than success rate and companies do not believe 
they can pick or choose. 

2 

Money spent on two representations could have been used on SW-CMM v2. 


MLs misused; continuous representation will encourage organizations. 

8 

Preference is A representation. 


Test is a motivator and comparative yardstick. There is a place in the community for 
the test. 


Conformance with business practices myth that you do process improvement for 
process improvement sake. Process improvement models are used for business 
drivers, and you need to find the quickest way to recover these dollars. 


3.5 Emphasis on Product Life Cycle 

Table 7 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Emphasis on Product Life Cycle.” Column two above the “Discussion” bar contains the 
results of the brainstorming. Column two below the “Discussion” bar contains related points 
made during participant discussion of the issues. Column 1 contains the number of votes for each 
issue/discussion point that indicates its importance to the workshop participants. 
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Table 7 Emphasis on Product Life Cycle 


Votes 

Issues 

Brainstorming 

8 

Interpreting for maintenance environment is difficult 

4 

CMM1 is not only applicable to development, AND it addresses the full lifecycle, and 
this is a good thing. We do a lot of work maintaining code written by someone else, 
and CMMI helps. 


Haven’t heard of big product companies using CMMI. Many of requirements are non- 
applicable. 

5 

Product development is one of the areas where there is the most change. Process Areas 
like Technical Solutions, Product Integration, Decision Analysis and Resolution, Inte¬ 
grated Teaming, and Organization Environment for Integration would find maximum 
applicability in the context of product development. Hence, need to do pilots to see 
results. 


[To quote George Box,] CMMI is wrong but is very useful; judgment is required. 

Could be “overkill” in detail. 

1 

Most people in SW-CMM were familiar with software, now there is an increased like¬ 
lihood that we won’t understand material—examples are needed more than ever in 
CMMI. 

5 

The CMMI addresses full life cycle of the product, which benefits both commercial 
software and IT/IS organizations by helping them to manage application portfolios 
across a long period of time. 

8 

Literal interpretation of SW-CMM was an issue. Literal interpretation of CMMI is an 
issue. See issue a: The CMMI can be interpreted to focus on new product develop¬ 
ment, where most IT/IS organization workloads are based on existing product en¬ 
hancement/maintenance, causing resistance to usage of the model as a tool in process 
improvement. 

Discussion 


We don’t do new products, so how do you interpret this practice. 


Lip service about full lifecycle, cradle-to-grave, still emphasizes development. 


Make/buy/reuse, evaluating alternatives, TS too much to ask when doing a 40-hour fix. 
Okay, [they’re] not applicable. How do I interpret this? Not do practice, do something 
else, or ... 


There is an extreme—either literal interpretation or do nothing. 
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Votes 

Issues 


The way that organizations define and treat what is a product and what is a project is 
different in this workshop. 


Sketchy product life cycle: 

not applicable 

consider project full life cycle 


Example of a layered situation. 


What is the “world” of the typical user of CMMI—what does it look like? 

8 

Need tailoring guidelines and examples—not just saying it is non-applicable. For 
example, how do you tailor the PAs if you have COTS, are doing maintenance and not 
new development, etc.? 


Identify types of work: 

new development 
adding pieces (project) 
quarterly release 
smaller than projects 
patches 

infrastructure projects 

engineering projects (e.g., database systems, new operating systems) 


3.6 Assessments 

Table 8 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Assessments.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 


Table 8 Assessments 


Votes 

Issues 

4 

More complex because of scale of model, not method, cost of assessments concern. 
Having one method is good. 

7 

Partitioning of appraisals, more alternative appraisal methods to assist with transition. 
(Another participant states common appraisal framework fCAFl allows this.) 

1 

One of the reasons we got away from continuous—difficulty in assessing it and then 
communicating results. 
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Votes 

Issues 

10 

Due to the expanded scope, demands other approaches for doing appraisals. 

• delta appraisals 

• multi-phase appraisals 

• surveillance appraisals 

7 

SCAMPI vl.l has process improvement indicators (PIIs) to help reduce burden of 
collecting info. SCAMPI Vl.l is very different from version 1.0 (which was almost 
identical to CBA IPI). Comfortable to do ten-day assessment on all areas (think 
organization is ML5); targeting 100 hours. Price to pay is to go to verification 
approach that is close to an audit approach. Before we get on site, the organization has 
to fill out several PII forms (significant internal effort prior to onsite). 

8 

Skill issue, so much content in model (systems engineering and software, staged and 
continuous, IPPD). This seems like a whole lot of material for the lead appraiser to 
know. 


Cost, some of the cost burden is moving within the organization, and now because of 
PIIs may spend two days where we spent four hours. 

6 

Training of LAs and quality issues. Greater understanding of model and greater 
control of quality, which will cost more. 


Approach and needs of a low maturity organization may be less than those of a high 
maturity organization. 

2 

Not clear that an appraisal done in verification mode meets the objective of the 
diagnosis mode of the IDEAL SM model. Especially true in low maturity organizations. 

6 

Strongly support delta appraisals. 

• PIIs were created on wall charts during on-site period, now they are done 
beforehand. 

• On-site period kept to < 100 hours. 

• Every time you do an appraisal start from scratch —> SCAMPI addresses this a 
bit with PIIs. 

• Learn how to use SCAMPI for process improvement. 


Large part of success of SW-CMM was due to level numbers in procurement 
language. People often adopt things for the wrong reasons, but are glad that they got 
to this new state. 

1 

Support level number, many of the internal initiatives we started to calibrate to Levels 
1-5. 


Write procurement language without level numbers. 


SM IDEAL is a service mark of Carnegie Mellon University. 
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Votes 

Issues 

5 

Budget burn rate issue—a little now, a little later. No guidelines for 
stretching/interpreting assessment requirements for CMMI (ARC). 

3 

SCAMPI does not provide a sampling method (failure), we just say go sample. 


3.7 Lack of Data 

Table 9 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Lack of Data.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 


Table 9 Lack of Data 


Votes 

Issues 

Brainstorming 

5 

Collect data on those assessed using continuous representation. 


Not clear if investment in SW-CMM is preserved and then enhanced. 

2 

Nice to have more granular data e.g., particularly regarding ROI continuous/staged, L2 
/ 1.3 / .... 

6 

Business decision—Do we want to spend money on SW-CMM or CMMI? ROI data 
on both SW-CMM and CMMI (comparison). 

2 

Wealth of data for L4 organizations. Show delta between SW-CMM and CMMI. Data 
should be available; actively solicit. 


Not aware of data available, gain of going from SW-CMM to CMMI. Sensitive to the 
maturity level the organization is at when transitioning to CMMI—and what is the 
“dip.” Data where systems perspective is strong versus other companies where 
systems are weak. Comparability when parts of CMMI are non-applicable (ROI). 

1 

Validation is a key issue when reporting ROI. Guidelines are needed for reporting 

ROI—not comparable and often times not valid. 


Unfair to expect data today, but need to put infrastructure in place to begin to collect 
data. (My expectations are currently low.) 
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Votes 

Issues 

10 

Data is across the board and retrospective. Need to provide a framework for collecting 
and reporting data. 

• demographics 

• starting performance 

• type of data 

• what do we mean by cost/investment 

2 

SEIR is okay. But rather than single organization to collect. It is more important to 
collect the right data. 

7 

Most data will be from early adopters. More valuable to use data from subset of 
organization. Comparison of data is a problem—need framework. Collection of data 
is possible. 

7 

Data related to procurements would be useful. Number of procurements, types of 
procurements, what kind of language and where. Procurement agencies recognize 
value of SW-CMM; now recognize value of CMMI. To republish data on SEI website 
is a good thing. 

2 

Guidelines for measurements being reported. Measurements reported are very creative 
in a negative sense. 

1 

SCAMPI should actively solicit amount of time it takes to complete PIIs (over and 
above normal). 


Fear SEI uses data in a self-serving nature. 1994 report attributed to CMMI. 

5 

Unfair to have data, unfair to expect organizations that do not have mandate be early 
adopters. 

2 

Continuous representation lack of data regarding CLs. Need elaboration, examples, 
and typical work products for every PA. Need to separate material so that it is 
associated with specific CLs. It is a more complex approach in the continuous 
representation to define up-front the GGs and GPs for CL 3-5 and then repeat those in 
each PA without giving detail (via elaboration, examples, and typical work products) 
of what those GGs and GPs really mean for a given PA. 

2 

SW-CMM did not have other things in marketplace. Now there are CMMI and SW- 
CMM. Share piloting data with community. Ease of getting information from SEI. 

Need comparison between CMMI and SW-CMM. Different business decisions from 
late 1980’s to now for organizations. (There is more scrutiny today.) 

9 

Collaborate with USC/CSE with COCOMO model for CMMI. Sometimes easier to 
get qualitative data from executive workshops. 
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Votes 

Issues 

1 

Reinforce ROI requests are a resistance to change (when data is not available). 
Sometimes just arrange a meeting on the golf course. Sometimes asking for ROI when 
no baseline data is available is a game from senior management. We request data, 
what is in it for them? Be careful because you could get garbage. What data is much 
easier to get? Maybe “1 use CM MI” is just a start. Not talking about how they use it, 
just a gross statement to get started—easy solution. In other words, qualitative 
information from early adopters is an easy solution to put in place. 

10 

Argument on ROI data without thinking about ROI model. Do I want to go to ML 4 & 

5 in SW-CMM or do I want to go to CMMI? Business ROI model should be built to 
be a predictor. Common terminology and syntax. 

Discussion 


Bucket of things that get dollars 

• mandates 

• everything else —» this competes against everything based on a business case 


Vast majority of business came from commercial (1990) not the DoD. Commercial 
companies (banking, insurance) need computers to survive. Defense contractors view 
this as overhead. Early adopters were commercial organizations driven by need. 

Defense contractors didn’t come on board until the policy went into place. 


Experience was quite opposite. When there was published data, then the commercial 
organizations came on board. 1994 was when these companies came on board—once 
data was published and available. 


TR23 -» CMM -» CMMI 

i i 

Evolutionary Revolutionary because it has a competitor: 

SW-CMM 

Need to show there is more to offer. 

4 

Getting data is very simple—one example, ask project leaders, “Are you glad you had 
QA?” 


Information passed within peers of the company carries a lot of weight. 


Question of perception. SW-CMM is a competitor of CMMI—like saying teenager 
competitor of adult. This is an evolutionary path. You don’t have a choice to become 
an adult. 

2 

ROI data is mixed and matched and spun however they want. $15M-profit, $14M 
spent on CMM, now the $14M includes everything (training, executive meetings). 
Framework is needed. 
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Votes 

Issues 


With SW-CMM, very few alternatives. CMM, CMMI are viewed as either/or. 


Extreme programming (XP) process improvement is out there, and it is a non-trivial 
paradigm. Sells extremely easily to development groups. ROI on XP vs. ROI on 

CMM. 


Characterize the competitors 

SW-CMM CMMI 

i 

good old SW-CMM 
upgraded to Draft C 


Expect it will be more difficult to upgrade SW-CMM v2C than to adopt CMMI. 


3.8 Investment/Cost 

Table 10 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Investment/Cost.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 


Table 10 Investment/Cost 


Votes 

Issues 

Brainstorming 

4 

Lower level of entry costs for assessment should be established S27-40K. Establish 
lower skills but be authorized—coursework leading up. ($5K to do entry level Class B 
assessment.) 


Real competition [is] with prescriptive methodologies like Rational Unified Process 
(RUP) and XP rather than CMM/CMMI. 

5 

Cost is too high, is ludicrous. We need quality and higher level of understanding. 

$15K to become CMMI Instructor + SCAMPI Lead Appraiser™ (LA) + travel + lost 
opportunities—didn’t need observation. Did not pay observation cost for becoming 
instructor. Observation —» $1500/day + travel. Business model [is needed] to show 
that cost is worthwhile. 


SM SCAMPI Lead Appraiser is a service mark of Carnegie Mellon University. 
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Votes 


Issues 


15% royalty, fixed fee for internal. Phenomenal difference between becoming 
SCAMPI LA and CBA IPI LA. Need to be able to [do] most formal before doing least 
formal, so shouldn’t offer a Class B entry and fee. Cost is high, but value is there for a 
consultant. Might not be true for internal use. 


I don’t think it is going to cost $15K (look at supplementary chart). ISO auditor is 
$1500 and if TickIT—two observations and test/interview $5K. A LOT LESS—an 
order of magnitude less. ISO has exact same issue of consistent results. 


+ travel + [lost] opportunity costs 



USA 

International 

Intro to CMMI 

$1,500 

$3,000 

Intermediate CMMI 

2,500 

3,750 

SCAMPI LA 

4,000 

6,000 

Observation* 

15,000 

15,000 

Instructor Training 

7,500 

11,000 

Observation* 

4,500 

4,500 


$35,000 

$43,250 

Maintenance fees 

6,000 

9,000 

(for two years) 

$41,000 

$52,250 

Royalties 

Internal 

$125 

$187.50 

External 

15% 

17.5% 


[The following text was added after the workshop.] 


Instructor Training and Obsen’ation are optional, depending on whether the Lead 
Appraiser also wishes to teach CMMI courses. The requirement for SCAMPI LA 
obsen’ation may also be waived. The range for USA costs is therefore $29,000- 
$41,000, and the range for International costs is $36,750-$52,250. 


Path to consistent results is Intermediate course + observation + quality assurance 
program—and that may not be needed. 


Data is there, fact is there [that it is costly to move to CMMI], this is the reason for 
slow adoption —> retraining of personnel. Major challenge. Reasoning is very difficult 
to justify. 


Large number of LAs (for SW-CMM in company) [—10] cannot be grandfathered any 
more. 


Believe that grandfather period has been extended. 


3 


Difference in business model between cost for internal organization and consultant. 
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Votes 

Issues 

5 

Support additional qualifications of LAs. Costs are negligible for multi-million dollar 
company. Can we preserve cost of SW-CMM when going to CMMI? Need to 
preserve the costs and have positioned the process improvement efforts in this way. 

This needs to be emphasized by SEI. 

4 

LA costs are significant. 22,000 employees in company. Authorizing people and 
conducting assessments is the big picture. Royalty fee is not acceptable for something 
that is in public domain. Can there be a VOLUME discount? Release in public 
domain and then get what we want. Don’t understand why international costs are 
different; skills are the same. 


Additional fee is arbitrary. It was reduced by 25%, which is nice but.... Asking for a 
more important fee to Canadian citizen may violate the [North American] Free Trade 
Agreement. And even if it didn’t, is this decent? Is this a good approach to spread 

CMMI worldwide to treat non-US citizen differently? 


Do a lot of business internationally. 

2 

What is the SEI business model? Cost of tools and technologies to support. 

1 

Framework for next 10 years. Doesn’t look like there is a good handle on this. 

7 

Cost is an issue. SW-CMM caught up especially in India. Importance of 
understanding market and having policy that promotes use. Companies outside US 
have process-driven, quality. More and more off-shore development. Rules of game 
today need to be reconsidered to look at marketplace. 


$187.50 mav be unrealistic based on what can be charged AND we are promoting 
knowledge. Could not present this to management because would be thrown out. 

1 

Generally, fee structure isn’t a problem. Pay $50K for consultant to conduct 
assessment, so if that is what it costs me okay. Agree with internal capability versus 
consultant—would like to see a different model. 


Didn’t pay all the fees because I traded the SEI services for fees. Paid ~ $25K US. 
Multiply by 1.5 for Canadian conversion. Salary conversion in Canada is not 1.5 and I 
really feel what it could be for the folks of India. Felt I needed the high maturity 
course also. HUGE amount of money—but I restate that it was WORTH the money. 

If we treat all citizens as world citizens (equally), I think the price is less an issue (we 
get what we pay for). Not fair to compare to ISO. (Look at page count [difference 
between ISO and CMMI].) 
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Votes 

Issues 

3 

Additional tax put on foreigners is a shame, unfair competitive advantage. SEI’s legal 
advisors might have told the SEI they could charge more; it does not mean they must 
charge more. Other things are put in public domain, and access is equal for all. Why 
not do the same with an INTERNATIONAL model? Total price tag makes it look like 
you start from scratch—should have provided something to show evolution. 


$10K/yr, $ 150/student (SW-CMM). Dropping contract because can’t justify. Costs 
more than sending students here (SEI). 60 students required to break even. 

9 

Difference between developing internal and consultants. Three LAs + four candidates. 
Candidate lead assessors are still candidates because our company is not performing 
enough assessments. Because of costs, won’t go to CMMI. Sponsoring candidate lead 
assessors is just dumb [idea] when we aren’t doing enough assessments, but there is 
personal growth. No business case to go to SCAMPI. 

1 

Assume there is a business model that targets how many people we want to move to 
CMMI and then do we have the resources to train the target. 40 Intro to CMMI 
[transition partners], 4X SW-CMM. 


Demand for SCAMPI is zero because only have early adopters. 


Blind numbers in the abstract. Costs look like not enough or too much. The individual 
cost/engineer to process improvement should be X. SEI needs a flow of revenue to 
continue this program. Cost of CMMI, no different than cost of using CMM. 


Is this a worthy investment? In a $3 billion program this is a worthy investment. 
Investment is well worthwhile. 

2 

Unfair advantage is incorrect because of the federal taxes that US organizations pay to 
do business. 


Understand difference between the US and those who deal with the government. 


• Government agencies aren’t willing to pay for this on their own. 

1 

• Tremendous inconsistencies among government agencies and therefore can’t put 
program in place to recover costs. 


• Contracts are still granted based on cost. 


• If the ultimate goal is increase adoption to CMMI, then lower the barriers 
(costs). 


• SW-CMM entry costs were lower than CMMI (perception). 
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Votes 

Issues 


• ISO 9000 is an international standard. Because of the standardization, you get 
more consistency. 


• In CMM, the cost of assessments appears to be whatever you can charge and get 
away with. ISO usually has about 10% variation between proposals/bids. 


SW-CMM 2% market penetration in ten years. Doesn’t sound like run-away train. 

Cost substantially influences business organizations. Numbers in SEI database are 
from same companies trying to make inroads into the SW-CMM community. Can’t 
get any easier than this. Price of entry is substantially higher 

• increased scope 

• increased training costs 

8 

Much higher than SW-CMM. Need approach to establish market share. We need 
other restaurant approach. 

Discussion 


Number of organizations at ML 3-5 is far more in India than any other country. COST 
is important (tried to convince other India organizations to come to workshop). 


Pricing—have to look at other factors in market (cost of living). 


CMMI too costly for India. Then you have it done by someone who isn’t as qualified. 
You lose because you get not $1, let alone $187.00. 

4 

CMMI is not an American model but an international model. It’s in the public domain 
and the target is the world. Don’t charge more to go to museum... What is the logic? 
This needs to be revisited again. 


36% tax rate of US organizations—US government didn’t pay for Disneyland, exhibits 
for museums. There is a cost model that the SEI uses. Don’t have to be an instructor 
or attend training, but there is a decision. You can pick the model up for free. 

12 

Decision to extend grandfathering was a good thing but should be extended throughout 
the transition time. 


3.9 Terminology 

Table 11 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Terminology.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 
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Table 11 Terminology 


Votes 

Issues 

6 

Some terminology in CMMI is foreign to commercial IT. For example, the word 
“systems engineering” is often used in IT non-CMM terminology to refer to systems 
programmers (e.g. mainframe assembler language programmers) or others who are 
involved in acquiring and maintaining system software, i.e. operating system, security 
software, etc. Instead of continually adding to the end of CMMI title, e.g. CMMI- 
SE/SW/IPPD, why not simplify the title of the model, e.g. “CMMI for IT” or just 
“CMMI”... 

5 

Terminology stinks in both CMM and CMMI—let’s get over it. QA is testing. Word 
“oversight” in complicated. 


3.10 Limited Coverage 

Table 12 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Limited Coverage.” Column two above the “Discussion” bar contains the results of the 
brainstorming. Column two below the “Discussion” bar contains related points made during par¬ 
ticipant discussion of the issues. Column 1 contains the number of votes for each issue/discussion 
point that indicates its importance to the workshop participants. 


Table 12 Limited Coverage 


Votes 

Issues 

2 

I know an organization in France that is working on safety practices—propose as 
another slash in the name of CMMI (additional discipline). 

8 

L4 & 5 so sparse because few companies and now wanted more PAs. Expected more 
results at L4 and L5. L4 and L5 very inconsistent. Further insight into L4 and L5 is 
needed. 

6 

Pure IT stepped on slippery slope. CMMI expanded scope and benefit from change is 
low. 

10 

What are criteria for inclusion and exclusion of material (PAs) in CMMI? Should 
there be additional criteria to look at this further? There should be a clearly defined 
strategy developed by the SEI and stated in the CMMI for the criteria used to 
determine what is included and excluded from the model. To date, this criteria has 
been determined by DoD based on the existence of previous models that needed to be 
merged, e.g., SW, SE, SA, IPPD. Going forward, the SEI should consider other 
guidelines and criteria, e.g., security management and business objectives. 


L4 and L5—comparison of vl.l SW-CMM and CMMI vl.02—little difference. 

3 

Point to [more on] maintenance. 

7 

Malcolm Baldrige Award was not a source model and fully Vi the [Malcolm Baldrige] 
points go against the results of your actions. There is a place for business results in 
CMMI, and this will eliminate the competing dollars. 
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3.11 Training 

Table 13 contains the flip chart notes from the brainstorming of issues and discussion of the issue 
category “Training.” Column two above the “Discussion” bar contains the results of the brain¬ 
storming. Column two below the “Discussion” bar contains related points made during 
participant discussion of the issues. Column 1 contains the number of votes for each 
issue/discussion point that indicates its importance to the workshop participants. 


Table 13 Training 


Votes 

Issues 

19 

Do we have enough systems engineering experience under the belts of those training 
and assessing? 


Opportunity to do cost management. Different types of SCAMPI auditors—maybe 
certain Paining they don’t need. 

20 

Don’t have enough Paining experience among those Paining and assessing. Build 
training skills. 

8 

Training your entire tech staff on CMM1 doesn’t sit well —» what are the guidelines? 
Could be Catch-22 if this is perceived as necessary. 


Though only the process specialist needs to undergo training in CMM, all the people in 
the organization are required to undergo awareness training in CMMI. 


Initially believed that you only train on processes (QMS) but then had problems. 

When preparing for assessments, we found this was problem—that they didn’t 
understand what was being talked about. 

6 

Partition training against ML achievement, domain specific (designers....). 
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4 Brainstorming on Next Steps and Recom 
mendations 


An “affinity group” discussion was held, based on the list of issues identified as important. All 
issues with three or more votes were discussed. Table 14 contains the flip chart notes from the 
brainstorming of next steps and recommendations. Column 2, “Issues,” summarizes the issue dis¬ 
cussed. Column 3, “Recommendations,” summarizes the recommendations made by participants 
to address the issue. Column 1 contains the number of votes for each set of is¬ 
sues/recommendations that indicates its importance to the workshop participants. 


Table 14 Next Steps and Recommendations 


Votes 

Issues 

Recommendations 

6 

Decision to extend grandfathering was a 

Don’t overlook those who lost the 


good thing but should be extended 
throughout the transition time. 

Huge amount of LAs [for SW-CMM in 
company] cannot be grandfathered any 
more. Believe that grandfather period has 
been extended. 

opportunity to be grandfathered. 

6 

For software-only how do you interpret 

Develop tailoring guidelines for different 


CMMI consistently, if tailoring is left to 

contexts 


individuals’ wrong approach—need 
tailoring guidelines. 

• software-only 

• maintenance 

3 

Need tailoring guidelines and examples— 

• COTS products 


not just saying it is non-applicable. 

Meta-comment: Some items did not get 
any vote, but provide possible solutions to 

2 

Interpreting for maintenance environment 
is difficult. 

these items. 

5 

No compelling need. Have SW-CMM and 

If SEI cannot sustain SW-CMM, 


it is working and you can’t take it away. 

license/give away to party who can: 


Worst Case: have to develop an appraisal 

• at minimum v 1.1 

• release v2 

• update v2 


method. 
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Votes 

Issues 

Recommendations 

4 

Not just method issue but also model, 
continued maintenance of SW-CMM 

Promote SW-CMM to ISO status. 


(CMM V2) for organizations that are 
predominately software. 

Sponsored by ISRI. 



Allow the transition/sunset to be 
determined by the market 



• only transition if clear need 



Interpret CMMI for software. 



Apply SCAMPI to SW-CMM. 

4 

Due to the expanded scope, demands 

Solution to other approaches for doing 


other approaches for doing appraisals— 

appraisals helps address solution to 
critical information for CMMI 


• delta appraisals 

• multi-phase appraisal 

• surveillance appraisal 

compliance issue. 

2 

If I only assess 80% and say I’m CMMI 
compliant then I’m leaving out critical 
information, not truthful to customer. 


4 

Data is across the board and retrospective. 

Need to provide business ROI model to be 


Need to provide a framework for 
collecting and reporting data: 

predictor 


demographics 
starting performance 
type of data 

Guidelines for validating the data are used 
for ROI and ensure comparability. 


Guidelines for collecting and analyzing 


What do we mean by cost/investment? 

the data. 

3 

Argument on ROI data w/out thinking 

One ROI framework should support both 


about ROI model. Do I want to go to 

models SW-CMM and CMMI to allow 


ML4 &5 in SW-CMM or do I want to go 
to CMMI? Business ROI model should 

for valid comparison. 


be built to be a predictor. Common 
terminology and syntax. 

Guidelines for two roles: 



• organizations who provide ROI 

1 

CMMI more holistic approach—1) huge 

data 

culture change 2) transition—not having 

• SEI to take data 


proper roadmap 3) roadmap 4) leverage 

Asking SEI to play data steward role may 


investment —> solve and adoption will 

be wrong priority. 


occur. 
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Votes 


Issues 


Recommendations 


Business decision—Do we want to spend 
money on SW-CMM or CMMI? ROI 
data on both SW-CMM and CMMI 
(comparison). 

6 For software-only how do you interpret 
CMMI consistently, if tailoring is left to 
individuals’ wrong approach—need 
tailoring guidelines. 

1 MA is good with expanding business 

imperatives—this needs to be emphasized 
and expanded even more. 


Previous point is not a binary decision; 
perhaps SEI can do something minimal 
(ROI) of good benefit. 


“Competing numbers” —» “Competition 
between business development dollars 
and process improvement dollars” 

Commercial community is less contract 
driven and more market driven. Process 
improvement associated with (driven by) 
business goals is better. For example, 
exploiting continuous representation is a 
possible approach/solution 

Malcolm Baldrige is "business 
excellence” model for given company 
versus SW-CMM/CMMI is development 
model; latter is subset of former. 

Specific suggestions toward broadening 
SW-CMM/CMMI: 

• Add business goal-driven metrics 
to model 

• Or forthcoming (needed) SEPG 
guide 

"Business excellence” is unnatural fit in 
model (oppose previous point). 

Subcategory 7.4 of Malcolm Baldrige = 
internal results from process management; 
category 2 links to much of what you do 
for process management and human 
resources. CMMI partially covers 
category 5 (also People CMM). 

CMMI fits much better in Malcolm 
Baldrige framework. 

Clarification: point of 10 is not limited to 
Malcolm Baldrige (but get business 
success/value is). 
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2 


Don’t have enough training experience 
among those training and assessing. 
Build training skills. 


There are existing requirements for 
training experience for LAs and model 
trainers, they should be enforced and 
verified. 
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5 What Would Make You Transition to CMMI 
VI.1? 


To summarize the issues captured during the workshop and to understand the most critical issues, 
the question was then asked: “What Would Make You Transition to CMMI vl .1 ?” Every work¬ 
shop participant was encouraged to address this question. Table 15 summarizes the discussion. 


Table 15 What Would Make You Transition to CMMI? 

What Would Make You Transition to CMMI...? 


If I had ROI from transitioning SW-CMM to CMMI, that demonstrates positive benefit (even if 
it isn’t the prettiest data). 


A story that shows benefit (might be qualitative versus quantitative). 


Establish incremental roadmap—a path from SW-CMM to CMMI. 

• Acknowledge SW-CMM is a [means] to superset CMMI. 

Lower level of entry cost. 

• (reuse investment) 

Difficulty is not the issue, but applicability [of PAs or goals] is (and uncertainty of how LAs will 
judge applicability versus not applicable). 


If official tailoring/interpretation guidelines are available to address this. 

When I reflect on clients in terms of what causes them to willingly make a transition: 

1) Three things: Knowledge (knowledge of model and how used), understanding of how it 
will help organizational performance, and time to make the transition. 

2) Remember: We’re still in the early adopters’ phase. 

3) And: Some will say “over my dead body”—nothing will matter. 

What will drive one to CMMI: 

• Customer or senior management—dissatisfied (pained) at something that is addressed 
(better) by CMMI. 
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Would ease the transition: 

• Need to understand why SEI costs the transition mechanisms the way it does. 

• Above would ease the transition. 


Organizations need time to look at CMMI (agree with knowledge, understanding, time) 


Give the market the time. 


(CMMI is international venture) 

• Increase accessibility to training (LA, Intermediate, and Train-the-Trainer) for transition 
partners; revisit tax. 

SEI should ask its own IT organization: what will cause you to transition to CMMI? 

1) interpretation guidelines 

2) data on benefits 

3) time needed to understand and consider 
(above a partial summary) 


One scenario; if (CMMI were) mandated by customer/contract, this would speed adoption: 

• Downside: acquisition organization then de-scopes process. 

• Some commercial customers do ask for ML. 

• A DoD mandate would drive adoption some. 

• A pro/con to interpretation guideline: —> it helps but can be bible. 

There are always issues with a model. Some of them may be good issues (should be fixed). The 
real issue is can they be worked ? Or, does it need to be revised? 
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6 Conclusions 


The SEI and ISRI are taking all of the feedback from this workshop into consideration in deter¬ 
mining how to best support the process-improvement community. A number of different perspec¬ 
tives were represented by the workshop participants. 


There was considerable discussion (and disagreement) about the need for a software-only model. 
Possible software-only solutions included (1) maintaining the Software CMM indefinitely, (2) 
creating a Software-only version of CMMI, and (3) building CMMI interpretation guidelines for 
software-only organizations. 


Some participants felt strongly about the need for defining a strategy and criteria for what should 
be included in future releases of CMMI. Examples suggested included security management and 
business success factors. 


Based on the workshop discussions, the SEI has identified four main tasks it will pursue: 

• providing interpretation guidelines for specific types of organizations (e.g., software) 

• soliciting data on benefits, including both quantitative and qualitative data 

• providing time for the community to understand and consider CMMI, since it is early on the 
adoption curve 

• revisiting the business model for CMMI transition 

The issues and suggestions from this workshop will be analyzed and combined with issues and 
suggestions received from other areas of the community. The SEI desires to eliminate as many 
adoption issues as possible to make the transition to CMMI easy for all members of the process- 
improvement community. 

In turn, ISRI will consider the workshop discussions when it is developing collaborations with the 
SEI and undertaking efforts to sustain process-improvement initiatives for the global software 
community. 


We would once again like to thank everyone who submitted a position paper and/or attended the 
workshop. Your inputs will help us to understand what we must do to continue to support soft¬ 
ware process improvement. 
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